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ABSTRACT 

This catalog, intended for inservice prograa* 
directors, describes a large nuiber of activities for evalv.dting the 
impact of teacher inservice workshops on learner gains. The first 
description explains how to provide an activity-by-activity 
evaluation measure that identifies objectives, time period, 
resources, personnel, and success of outcome. The second describes 
how to identify positive and negative factors through an overview of 
the program. The third advises project directors to become aware of 
current literature on management and leadership by objectives. The 
fourth examines the problem of motivating all participants in an 
inservice program to strive toward new professional competency. The 
fifth considers the importance of being able to count visible and 
measurable success. The sixth looks at what can be done to prepare 
for local implementation difficulties. The seventh discusses followup 
evaluation instruments. The eighth describes how to develop an 
instrument to determine the competence level of participants at the 
beginning of the workshop in relation to the workshop's objectives. 
The ninth discusses participants' comments on and responses to the 
workshop. The tenth explains that the objectives proposed for the 
workshop should be realistic. The eleventh stresses the importance of 
considering what the participants want; and the last description 
emphasizes the fact that this catalog is not a blueprint telling the 
director exactly what to do, but a list of alternatives. (PB) 
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INTRODUCTION 



This dociMneat is a catalogue. This document provides a large number 
of evaluation activities that can be used to evaluate the impact of teacher 
inservice workshops on documented learner gains. 

A learner gain, sometimes called a learner benefit, is a measurable 
increase in knowledge, skills, or attitudes that students aquire as a result 
of participation by their teachers in teacher inservice workshops. 

This document should be seen as an outline more than as a prescription. 

Each individual application will stress only one or two of the learning 
environments suggested for evaluation of learner gains. 

Any reader interested in coming up with more detailed and more specific 
evaluation techniques should feel free to contact the author. Other 
publications giving exact derails are available. 
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EVALUATING THE IMPACT OF 

TEACHER INSERVICE WORKSHOPS 
ON DOCUMENTED LEARNER GAINS 



TITLE: Inservice Education Program Evaluation 

AUDIENCE: Project Director of Inservice Workshop 
OBJECTIVES: 

1. Provide an activity^by-activity evaluation that identifies 
objective, time period, resources, personnel, and success 
of outcome* 

2. Identify activities and organizational mechanics which 
contributed or detracted from the overall objectives. 

3. Measure: (a) the degree to which participants have or 

have not met the standards of acceptable 
achievement inherent in the program 
objectives. 

(b) the extent to which major instructional 
activities enhanced or detracted from 
program objectives* 

(c) the extent to which non-instructional 
program elements (library, materials' 
availability, location, management, 
housing, food, climate control, 
non-classroom interaction opportunities, 
registration procedure, stipend 
reimbursement) facilitated learning. 

4. Describe the followup envisioned in the next six months to 
measure (a) how competencies acquired, materials developed, 
or strategies planned were implemented, (b) the results of 
the implementation, and (c) the situation in and under which 
the implementation occurred. 

5. Report the results of the followup evaluation from individual 
teachers . 
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I BASELINE CHECKLIST 



Does the proposal do the following? 



State objectives in behavioral terms. 

State objectives in terms of participant outcomes. 

Specify terminal behaviors acceptable as 
documentation objectives have been met. 

Link each learning activity to one or more of 
the specific objectives. 

Provide a minimum of one learning activity for 
each objective. 

Measure overall reaction to the program from: 

(a) participants 

(b) staff 

(c) director 

Contain a form for the immediate program 
evaluation by participants, staff, and 
director. 

Include qualifications of all project staff and 
director. 

Specify time and site of ac*-ivities. 

YES NO 10. Indicate, if applicable: (a) the amount of collegiate 

Credit (graduate or 
undergraduate) 
(b) the differential requirements 

betx^een graduate and under- 
graduate credit. 

YES NO 11» Specify (if applicable) special institutional admission 

requirements. 

'ES NO 12. Include the following in the budget sunsnary: 

A. Name, address, and phone number 
of responsible financial office. 

B. The submitting agency's internal 
assignment number 

C. Budget summary notes numerically 
sequential to the budget summary 

D. Actual computation of staff salaries 
and consultant fees in relation to 
institutional policies. 
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YES NO I. 

YES NO 2. 

YES NO 3. 

YES NO A. 

YES NO 5. 

YES NO 6. 

YES NO 7. 

YES NO 8. 

YES NO 9. 



LEAKNING ENVIRONMENTS 



LEARNING ENVIRONMENT l i ACTIVITY-BY-ACTIVITY EVALUATION (OBJ-1) 

In order to provide an a ctivity-by activity evaluation that identifies 
objectives, time period, resources > personnel, and success of outcome, the 
following procedures may be employed. 

U Identify the major activities of the inservice education program. 

2. Attach to each major activity in the program a short list of two or 
three objectives. Each of these objectives should begin with a verb. 
The subject of each of these verbs should be the participant attending 
the Inservice education program. 

3. Provide an evaluation form that lists, among other items, the objective 
number and a chance to rate it from very good, good, average, poor, or 
Inferior. 

4. Make sure that the above procedures are carried out with adequate 
explanation provided to participants as to what is desired in this 
type of evaluation. 



This type of evaluation is done activlty-by-activity on a very detailed 
level. This type of activity will provide a number of counts that can 
provide a sesslon-by-sesslon evaluation. 



LEARNING E N^/IROl^lENT 2 i IDENTIFY POSITIVE Am l^EGATIVE FACTORS (OBJ-2) 

After the activity-by-activity evaluation stressed in Learning Environment 1, 
the program director Is able to form an overview. 

This overview requires some simple way of keeping score whether by 
percentage or by total points. 

For example, a typical overview could look like the following two columns. 
Column 1 is an identification of the session. Column 2 is an identification 
of the percentage of successful achievement of objectives. 



It Is the task of the project director to identify activities ot. organizational 
mechanics which contributed or detracted from the overall objectives. Looking 
at the two columns above provides a very* simple format to identify common 
factors that were either negative or positive. 



Session Identification 



Percent of Success 



Session 1 
Session 2 
S'ission 3 
Session 4 
Session 5 



95X 
60% 
40% 
90% 
30% 
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After careful analysis of the reasons for the outstanding success of 
session 1 and 4, it is possible for the program director to conclude that 
success was due to: 

Teacher personality 
Media adaptation 
Audience participation 
Question and answer period 
Open discussion 
Group interest in the topic 

Appropriateness of topics to the needs of the audience 

These positive coimnon factors should be stressed in future workshops 
as elements that contribute to success. These factors of success should be 
pointed out to the participants and instructors in order to reinforce the 
positive influence of these activities or organizational mechanics. 

In addition to finding out common factors of success, the program 
director shoufd look for weak links. This might mean that certain sessions 
such as 2, 3, and 5 were unsuccessful because of: 

Too much reliance on the lecture method 

A very dull presentation 

Nothing new was said 

The audience remained silent 

Audience needs were not addressed 

The topics covered were poorly presented 

After looking at these specific negative factors on an activlty-by- 
activlty evaluation, the program director is able to pinpoint specific 
things that should be eliminated the next time around. Sometimes, an 
activity-byactivity analysis can identify activities or structures that 
can be changed immediately on the spot. This on line Improvement will 
contribute to the overall success of the conference. 

Learning Environment 1 has stressed a MICRO approach to evaluation. 
Learning Environment 2 has stressed a MACRO approach to evaluation. The 
micro approach zeroes in on details. The macro approach tries to form an 
overview of the Inservice education program. Both elements are necessary 
for a well balanced evaluation. 
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LEARNING ENVIRONMENT 3 ; MA6fik (OBJ-3) 

It would be appropriate for project directors to become aware of some of 
the current literature on management by objectives and leadership by 
objectives, a few names would come to mind; Mdger> Popham, and others. 

These authors should be consulted either in textbooks or in journals In 
order to provide project directors with specific examples of how to 
evaluate a conference according to prespecifled objectives. 

One simple way to start would be to go to the ERIC collection and the 
attached RIE (Research in Education) Index. Looking up topics of Interest 
in this way would provide the project director with up-to-date information 
on a wide variety of approaches. After having looked at a wide variety of 
alternative activities and organizational mechanisms, the project director 
would be in a better position to tailor the workshop to individual needs. 



LEARNING ENVIRONMENT 4 : IMPORTANT ATTITUDES (OBJ-3) 



It is Important for the Inservice education program Instructor to realize 
that a small percentage of teachers present among the participants are there 
principally for academic credit or the small stipend. The inservice program 
imist be organized in such a way as to remotivate these teachers to participate 
to acquire a new competency. 

A new competency can be acquired and/or used on four levels within the framework 
inservice education programs: 

LEVEL I: The teacher acquires a new competency 

LEVEL II: The teacher applies this new competency back 

in the home school 
LEVEL III: The teacher adapts this competency to local 

implementation circumstances and difficulties 
LEVEL IV: The teacher documents the effectiveness of 

this new competency with specific examples of 

impact on student learning 

In this way, an inservice instructor can take teachers who are present for 
a wide variety of motives and remotivate these teachers into a striving 
after new professional competency. This is a challenging task, but the 
reward is worth the effort. 



LEARNING ENVIRONMENT 5 : COUNTABLE RESULTS (OBJ-4) 

One of the secrets of evaluation is to know what to count. 

This means that the successful participant in inservice education programs 
must be able to go back to the home school with a definite idea of countable 
student successes that document the importance of the teacher competency 
acquired during a workshop. 

For example, a teacher just ended a workshop on employability profiles. As 
a result of this workshop, the teacher was able to divide a printing course 
into five employability areas; 

1. Operating the offset camera 

2. Stripping the negatives 

3. Making the plates 

4. Operating the press 

5. Binding printed material 

With such an approach, the teacher was able to count partial as well as 
complete successes* A partial success would be an instance wherein a 
student was able to gain entry level employment In one of the five major areas. 
A conplete success will be an instance wherein a student was able to obtain 
entry level employment in two or three of the major offset printing jobs. 
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The ability to count visible and measurable success Is an important point 
in the type of foUowup necessary to measure the impact of inservice 
education programs. 



.LEARNING ENVIRONMENT 6: LOCAL PUNS FOR CUSSROOM TEACHERS (OBJ-5) 

Some teachers go to a workshop with the anticipation of coming home with a 
prefabricated plan that requires very little investment of local planning 
time. Such prefabricated plans do not always work out. 

The first step is for a teacher to acquire several examples of what 

has been found to work in a variety of individual circumstances. 

The SECoidstep is for the teacher at the Workshop to preview some of the 
first implementation steps that can be taken in light of the workshop period. 

The third step at the workshop is for the group to preview local Implementation 
difficulties. In some places, the schedule will be different, the students 
will be different, the barriers will be different, the problems will be 
different, and the reactions of the staff will be different. 

In anticipating the necessity for local adaptation, the educator at an 
inservice education program will be in a better position to overcome local 
implementation difficulties. ^ 

After this has been done, the teacher should try to come up with countable 
examples that document learner success dtie to competency possessed by the 
teacher. 



ERLC 



LEARNING EN VIRONMENT 7 i FOLLOWUP EVALUATION INSTRUMENTS (OBJ-5) 

It is highly desirable to find out what teachers did as a result of 
participating in inservice education programs. 

There are some advantages in using a single instrument to tabulate the 
followup results by comparing one teacher with another. However, room 
should be left to give each person enough freedom to use individual 
ingenuity* 

Sometimes, individual ingenuity is found expressed in a homemade followup 
instrument. The important point here is to make sure that the local 
followup instrument provides data that can be used by other educators 
in different schools and in different circumstances. 

One simple followup technique is to use a telephone survey. This enables 
the project director to contact a selected sample of workshop participants 
in order to ask such questions as: 

1. In what specific ways have you been able to use the 
knowledge, skills, or attitudes picked up at the 
inservice education program you recently attended? 

2. How have you been able to document learner success related to 
O competencies acquired at the recent inservice education workshop! 



LEARNING E NVIRONMENT 8 ; BASELINE DATA INSTRUMENT- (PRETEST-I-2-3) 



It is the responsibility of the project director to come up with some 
type of an instrument to pinpoint precisely where the participants 
are at the beginning of the workshop in relation to the objectives of 
the workshop • 

The gathering of such baseline data avoids the situation wherein most 
participants are exposed during the first few days to things they 
already know. 

To facilitate the best planning; such a baseline data check could be a 
duplicated instrument sent along with the application for the inservice 
workshop, but in any event prior to the conduct of the workshop • 

This, baseline data instrument should include knowledge evaluation (KE), 
performance evaluation (PE), and attitude evaluation (AE) . 

KE refers to things the participant already 
knows or has previously read about, 

PE refers to background experiences and 
present skill levels of the participants in 
the areas under study in the workshop, 

AE refers to sounding out the feelings 
and values of the participants with 
regard to the objectives, yardsticks, 
and procedures of the workshop. 

This kind of a check on where the participants are should be done several 
weeks or months before the workshop. This type of diagnostic preassessing 
enables the workshop director to plan a program that is best suited to the 
participants as a group or individuals depending on the activity. 



LEARNING E NVIRONMENT 9 t DIFFERENT EVALUATION PERSPECTIVES (PRETEST-6) 

Experience seems* to indicate that even the best workshop will evoke a nurber 
of positive and negative comments on the part of participants. Whenever 
the comments are skewed too much in either the negative or positive direction, 
something is out of order. 

Without going to either extreme, the negative extreme of pressing participants 
to find something wrong with an excellent workshop or the positive extreme 
of forcing participants to find something good about an inferior workshop, 
the evaluation should include both negative and positive observations. 

One simple way to do this is to stress at least three different perspectives. 

Perspective I could be the perspective of the wor* shop director and his 
objectives. This simply means that the workshop director keeps score and 
explains how his overall evaluation was arrived at. 
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The second perspective is that of the participants and their objectives t This 
simply means that each participant spells out the major anticipation for the 
workshop as well as the major results of this workshop when viewed from the 
individual's point of view. 

The third perspective is that of outside evaluators. ^ ^search and 

evaluation point of view, this would mean plnpointini . ,aat can be 
duplicated elsewhere at a reasonable cost with good e^^^w . cations of success. 
From a management point of view, this would mean deciding whether or not to 
conduct the institute the next year at the same site or at a different site> 
with the same workshop personnel or with different individuals. 

The workshop director who is aware of these 3 different perspectives as well 
as a number of other possible di cfering expectations is in a good position to 
provide the type of evaluation that documents the overall impact of the workshop 
in contributing to educational progress* 



LEARNING ENVIRONMENT 10 ; REALISTIC EXPECTATIONS (PRETEST-1-2-3) 

The objectives proposed for a specific workshop should be realistic 
expectations. This means that the budget, the time available, the 
instructional personnel, and the participants are able to accomplish the 
prespecified objectives in the plan put forward. 

As far as knowledge is concerned, two or three days should be more than 
adequate to convey the basic information and conceptual framework. 

When a workshop has objectives that go into the performance or attitude 
domain, two or three consecutive days are normally adequate to the task. 
Sometimes, this type of a program can be arranged to have two or three one- 
day sessions several weeks apart. The time between sessions allows both 
staff and participants to readjust individual activities in order to 
achieve prespecified objectives with greater success. 

The difference between a realistic expectation and an impossible dream depends 
upon a number of variables. A thorough awareness of the competencies of 
participants can have much impact here* Given the right project director, 
staff, and participants, almost any objective can be achieved if the budget, 
timetable, and resources are adequate. 

However, the typical situation is that only so much can be achieved in a 
certain period of time given the constraints of the participants and staff. 
A realistic expectation requires an accurate and objective assessment of what 
can be done in a given period of time. 



tEARi^'tNG ENVIRONMENT 11 SPECIFYING WHAT IS WANTED (OBJ-1) 



It Is p.r 'nes interesting to tale the view of an outside observer 
watchi different mentalities plan for an inservice workshop. 

One tyi>« mentality is constantly asking and answering the question, 
••What Is wanted!" This type of person zeroes in on objectives, purposes, 
and benefits. The results of such an inquiry are usually specific and 
measurable. 

A second type of mentality is constantly asking and answering in a 
dozen different ways the question, 'Vho is going to teach this workshop?" 
This type oi: person possibly has a pal or crony in mind. The difficulty 
arises from the fact that the well qualified associate may be si©r# on 
the mind of the planner than the needs of the typical participant for 
whom the workshop is being designed. 

There is nothing the matter with either of the above questions. On the other 
hand, the two questions are not equivalent. The t^^o questions are not of equal 
value* The first question to be asked in planning a workshop is, '*What 
is wanted?" The next auestion, which must come after the first question 
has been answered, is/'Who will teach in the workshop?" 

The practical advantage of asking the first question first is obviouly found 
In the ability to come up with a l&^ge number of answers for the second 
question If the objectives of the workshop are precisely spelled out. 
When a specific desired product of a workshop is clearly identified, 
the personnel Involved are able to produce this result with a number of 
different people, at a number of different price tags, in a number of 
different ways, and in a vatiety of different circumstances. Anyone 
who asks the second question first is like somebody who want^to go from 
city X to city Y but only by a specific road. If the road is chosen 
before the destination, it is quite likely that a new express highway 
will never be utilized by a planner who chooses the path before the 
destination. 



LEARNING ENVIRONMENT 12 DISTINGUISHING BETWEEN A CATALOGUE AND A BLUEPRINT 

(OBJ-1, OBJ-2) 

The term LEARNING ENVIRONMENT has been used in place of guidelines, 
criteria, or directives for the evaluation techniques herein suggested. 
A catalogue lists a large number of Items from which each individual 
will choose one or two appropriate tools. A blueprint lists everything 
that must be Included in order to make the construction complete. With 
these definitions in mind, this collection of learning environments 
Is more like a catalogue than a blueprint. 

As In pro football, this collection is like a play book. It Is a good 
list that gives a wide variety of alternatives. Each project director 
is to consider himself an evaluation coach who will choose one play 
or strategy at a time. The exact choice will be made to match the 
local team available and other individual constraints-. 



The Inexperienced project director will feel It obligatory to cover 
every learning environment in this collection. The hesitant project 
director might even feel it necessary to regiirgitate this catalogue 
of evaluation examples with only a few words changed to avoid total 
plagarism. Both of these errors boil down to confusing this catalogue 
with a detailed blueprint. 

This collection «nCourages project directors to thlak seriously 
about evaluation. The proof and result of this serious thinking is 
to be a documented plan worked out by each individual workshop director. 
This workshop director will spell out individual objectives and targets. 
In order to make evaluation more objective, each workshop' director will 
spell out a number of acceptable yardsticks with which to measure progress. 
Sowe yardsticks will be original, others will be borrwed, and others 
will be adaptations or combinations of successful strategies found 
elsewhere. 

With this perspective, any hypothetical case presented in any learning 
exrviromnent is to be interpreted as an exan^le rather than a« a 
specification. This example is intended to show what ctjuld be done 
In order to give a concrete example in place of abstractions. Obviously, 
the workshop director who would copy this example detail for detail 
would be manifesting a certain amount of incompetency since it is 
difficult to assume that local situations would exactly parallel the 
circumstances under which this example was developed. 
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Here Is where each reader writes in an Idea, strategy, or value 
not cited above* 



